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DETAILED ACTION 

1. A request for continued examination under 37 CFR 1.114, including the fee set 
forth in 37 CFR 1 .17(e), was filed in this application after final rejection. Since this 
application is eligible for continued examination under 37 CFR 1.114, and the fee set 
forth in 37 CFR 1 .17(e) has been timely paid, the finality of the previous Office action 
has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on July 29, 
2004 has been entered. 

Response to Amendment 

2. Applicant's amendment filed on June 4, 2004, has been entered and made of 
record. 

Response to Amendment 

3. Applicant's arguments regarding Claims 1,18, 35, 52, 66, 80 and 94 have been 
fully considered but they are not persuasive. 

Applicant argues that prior art of record (Modestino et al) do not disclose, 
"assigning one of a plurality of predetermined stereotypes to the labeled region 
adjacency graph (RAG) according to at least one identified predetermined pattern of the 
semantic labels in the labeled RAG, such that the assigned stereotypes describes the 
plurality of the digital image (i.e., the entire RAG and the entire digital image)". 

The Examiner disagrees and indicates that Modestino et al clearly indicates "In 
high-level processing, image domain knowledge is used to assign object label to the 
primitives, thereby providing a semantic description of the image. The joint 
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assignment of object labels for all the image primitives provides a stereotype 
interpretation of the image. 

This means a stereotype (high-level interpretation) is associated with the image 
semantic label. Modestino et al teachings clearly disclose this concept (Figures 1(a)- 
1(c), 5(a)-5(d) and 6(a)-6(d); Tables l-IV; Page 606, Section I, Introduction, Paragraphs 
1-3; Pages 610-613, Sections V-VII). Modestino et al disclose a Markov Random Field 
(MRF) model-based approach image interpretation and classification by performing 
knowledge-based high-level processing via assigning higher-level expressions 
(Stereotypes) to the classified objects. The region adjacency graph is classified based 
on higher-level expression or stereotypes (e.g., rural road scenes comprising road, field, 
car and sky). 

Modestino et al.'s teachings are not limited to "one label to one region". "Two 
regions" and "three regions" categories depicted in Table I in association of Figures 
1(a)-1(c) are stereotype labeling or higher-level expressions. 

It is further submitted that the invention outlined in the paragraph connecting 
Pages 31 and 32, is not recited in the claimed subject matter. 

Claim 94 recites "analyzing the labeled RAG by assigning one or more of a 
plurality of stereotypes". Consequently, with consideration of one stereotype, 
analogous arguments presented for Claims 1 , 8, 9 and 14, are applicable to Claim 94. 

Claim Rejections - 35 USC § 103 
4. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 
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(a) A patent may not be obtained though the invention is not identically disclosed or described as set 
forth in section 102 of this title, if the differences between the subject matter sought to be patented and 
the prior art are such that the subject matter as a whole would have been obvious at the time the 
invention was made to a person having ordinary skill in the art to which said subject matter pertains. 
Patentability shall not be negatived by the manner in which the invention was made. 

5. Claims 1, 3-10, 13-16, 18, 20-27, 30-33, 35, 37-44, 47-50, 52, 54-61, 66, 68-75, 
80, 82-89 and 94-96 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Modestino et al (IEEE Paper ISBN: 0162-8828; A Markov random Field Model-Based 
Approach to Image Interpretation) in view of Jain et al (US 6,360,234). 

Regarding Claim 1 , Modestino et al disclose a method of classifying a digital 
image, said method comprising the steps of: 

segmenting the digital image into a plurality of substantially homogeneous 
regions (Figure 1; Pages 607-608, Section ll-B. The image in Figure 1a is segmented 
into homogeneous nodes or regions, R = {R 1r R 2 , R n ) as depicted in Figure 1(c).); 

processing the plurality of regions to provide a region adjacency graph (Figure 1 ; 
Section II) for the digital image, the region adjacency graph representing adjacencies 
between the plurality of regions of the digital image (Figure 1(b); 

labeling at least one of the regions of the region adjacency graph with one of a 
plurality of predetermined semantic labels to provide a labeled region adjacency graph 
(Figure 1 ; Page 607, Column 2, Section IIB, first Paragraph. G = {/?, £} is the region 
adjacency graph comprising of the set of nodes R = {R 1t R 2 , Rn} and the set of 
edges £ connecting the regions. The labels assigned to the segmented regions. L = 
{L 1t L 2 , L M } is the set of all predetermined semantic labels.); 

analyzing the labeled region adjacency graph to identify one or more 
predetermined patterns of the semantic labels in the labeled region adjacency graph 
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(Abstract; Figure 1; Pages 608-609, Section lll-B. The image in Figure 1a is segmented 
into homogeneous nodes or regions, R = {Ri, R2, R n ) as depicted in Figure 1(c). 
Section NIB describes predetermined patterns of the semantic labels; e.g., the second 
type of clique basis functions for spatial constraints, "a car should be on the road" or "a 
car should never be in the sky. Predetermined patterns of the semantic labels are 
identified as shown in Tables l-IV, i.e., possible combinations and impossible 
combinations.); 

assigning one of a plurality of predetermined stereotypes to the labeled region 
adjacency graph according to at least one identified predetermined pattern of the 
semantic labels in the labeled region adjacency graph, each of the predetermined 
stereotypes corresponding to at least one of the predetermined patterns such that the 
assigned stereotype describes the plurality of regions of the digital image and 
represents a classification of the digital image (Figures 1, 5(a)-5(d), 6(a)-6(c); Page 606, 
Introduction; Pages 608-610, Section lll-B, Clique Functions for Multiple-Node Cliques ; 
Tables l-IV. The region adjacency graph is classified based on higher-level expression 
or stereotypes (e.g., " a car should be on (neighboring to) the road' is an example of 
stereotype to the labeled RAG; also rural road scenes comprising road, field, car and 
sky. "Two regions" and "three regions" categories depicted in Table I in association of 
Figures 1 (a)-1 (c) are components of stereotypes or higher-level expressions. The joint 
assignment of object labels for all the image primitives provides a stereotype for 
interpretation of the image. The digital image is classified based on the identified 
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patterns (features) of the regions as depicted in Figures 1 , 5(a)-5(d), 6(a)-6(c) and 
Tables l-IV.). 

Modestino et al do not explicitly disclose storing the assigned stereotype and the 
digital image in one or more databases of digital images, wherein the digital image is 
retrievable from the one or more databases using the assigned stereotypes. 

"storing the assigned stereotype (higher-level expressions) and the digital image 
in databases of digital images to retrieve the digital image from the databases using the 
assigned stereotypes" is the fundamental concept in "Content-Based Image Retrieval, 
CBIR" and "Query-Based Image Retrieval, QBIR" well known in the art (Official Notice). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al.'s teachings to retrieve the digital image 
from the databases using the stored stereotypes corresponding to these digital images 
because it is a well known methodology routinely implemented in Content-Based Image 
Retrieval and Query-Based Image Retrieval systems wherein complex digital images 
are retrievable by implementing higher level intelligible contextual information. 

Regarding Claim 3, Modestino et al further disclose the method according to 
Claim 1 , wherein identification of the predetermined pattern is based on a size of one or 
more regions of the digital image (Section III, first paragraph; Section 1MB; Tables l-IV. 
Classification is based on the areas and the boundary lengths of the regions identified 
by the RAG nodes.). 

Regarding Claim 4, Modestino et al further disclose the method according to 
Claim 1 , wherein identification of the predetermined pattern is based on an adjacency of 
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the regions (Figure 1; Pages 608-609, Section lll-B; Figures 1 and 6(b). Region 
adjacency graphs are inherently generated based on the spatial dependence of objects 
in the relative proximity of each other as depicted in Figure 1(c).). 

Regarding Claim 5, Modestino et al further disclose the method according to 
Claim 1, wherein identification of the predetermined pattern is based on semantic label 
content of the region adjacency graph (Figure 1; Pages 608-609, Section III). 

Regarding Claim 6, Modestino et al further disclose the method according to 
Claim 1, wherein identification of the predetermined pattern is based on a mean color of 
one or more regions of the digital image (Figure 1; Pages 608-609, Section III; Page 
613, Table lll(a), Average Gray Level of the regions). 

Regarding Claim 7, Modestino et al. do not disclose the method according to 
Claim 1 , wherein the plurality of stereotypes are stored in an association lookup table. 

Jain et al further disclose the method according to Claim 1 , wherein the plurality 
of stereotypes are stored in an association lookup table (Figures 6, 9, 16 and 17; 
Column 6, Lines 29-47; Column 13, Lines 52-67, Column 14, Lines 1-31). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al.'s teachings according to Jain et al. 
teachings to store the plurality of stereotypes in an association lookup table because it 
is a well known methodology routinely implemented in the art for information storage 
and retrieval. 
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Regarding Claim 8, Jain et al further disclose the method according to Claim 1 , 
wherein the stereotypes are represented in a hierarchal arrangement (Figures 7, 9 and 
17; Column 6, Lines 29-47; Column 13, Lines 52-67, Column 14, Lines 1-31). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al.'s teachings according to Jain et al. 
teachings to represent the stereotypes in a hierarchal arrangement because it is a well 
known methodology routinely implemented in the art for inter-relating image retrieval 
information. 

Regarding Claim 9, arguments analogous to those presented for Claim 8 are 
applicable to Claim 9. 

Regarding Claim 10, Modestino et al disclose the method according to Claim 1, 
wherein the region adjacency graph is provided by analyzing contextual data associated 
with one or more regions of the digital image (Abstract; Page 606, Introduction; Figure 
1 , Page 607-609, Sections MB and 3, first Paragraph). 

Regarding Claim 13, Modestino et al do not disclose the method according to 
Claim 1, further comprising the step of providing metadata associated with the digital 
image, wherein the metadata includes the stereotypes of the digital image. 

Jain et al disclose a method for video cataloging by providing metadata 
associated with the image comprising the step of providing metadata associated with 
the digital image, wherein the metadata includes the stereotypes of the digital image 
(Figures 16 and 17; Column 13, Lines 52-67, Column 14, Lines 1-31). 
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It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al invention according to the teachings of 
Jain et al to provide metadata associated with the digital image, wherein the metadata 
includes the stereotypes of the digital image because it will improve image appearance 
and enhance image classification. It will accurately classify and intelligently extract 
information, termed metadata, about the contents of video stream in real time (Jain et 
al, Column 1 , Lines 46-49) 

Regarding Claim 14, Jain et al further disclose the method according to Claim 13, 
wherein the metadata includes a hierarchical path associated with the respective 
stereotype of each digital image (Figures 9, 15-17; Table 1; Column 8, Lines 22-61). 

Regarding Claim 15, Jain et al further disclose the method according to Claim 14, 
wherein the hierarchical path is stored with a respective stereotype as a metadata 
object which is associated with a respective image object (Figure 6; Column 6, Lines 
29-38). 

Regarding Claim 16, Jain et al further disclose the method according to Claim 14, 
wherein the hierarchical path is stored as a referenced lookup table (Figure 7). 

With regards to Claims 18, 35, 52, 66 and 80, arguments analogous to those 
presented for Claim 1 are applicable to Claims 18, 35, 52, 66 and 80. 

Concerning Claims 52, 66 and 80, Modestino et al further disclose providing a 
set of labeled regions (Abstract Lines 5-10; Page 607, Column 2, Section MB, second 
paragraph, set of labels L = {Li, L 2 , L M }). Markov Random Field (MRF) model-based 
approach segments the image into a collection of disjoint regions that form the nodes of 
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an adjacency graph. Once the adjacency graph has been determined, image 
classification will be achieved through assigning object labels to the segmented regions 
using domain knowledge, extracted feature measurements, and spatial relationship 
between the various regions (Abstract, Lines 5-10). 

With regards to Claims 20, 37, 56, 70 and 84, arguments analogous to those 
presented for Claim 3 are applicable to Claims 20; 37, 56, 70 and 84. 

With regards to Claims 21 , 38, 55, 69 and 83, arguments analogous to those 
presented for Claim 4 are applicable to Claims 21 , 38, 55, 69 and 83. 

With regards to Claims 22, 39, 54, 68 and 82, arguments analogous to those 
presented for Claim 5 are applicable to Claims 22, 39, 54, 68 and 82. 

With regards to Claims 23, 40, 57, 58, 71, 72, 85 and 86, arguments analogous 
to those presented for Claim 6 are applicable to Claims 23, 40, 57, 58, 71, 72, 85 and 
86. 

With regards to Claims 24, 41 , 59, 73 and 87, arguments analogous to those 
presented for Claim 7 are applicable to Claims 24, 41 , 59, 73 and 87. 

With regards to Claims 25, 42, 60, 74 and 88, arguments analogous to those 
presented for Claim 8 are applicable to Claims 25, 42, 60, 74 and 88. 

With regards to Claims 26, 43, 61 , 75 and 89, arguments analogous to those 
presented for Claim 9 are applicable to Claims 26, 43, 61 , 75 and 89. 

With regards to Claims 27 and 44, arguments analogous to those presented for 
Claim 10 are applicable to Claims 27 and 44. 
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Regarding Claim 30, Modestino et al do not disclose the method according to 
Claim 18 further comprising the step of providing metadata associated with the digital 
image, wherein the metadata includes the stereotypes of the digital image. 

Jain et al disclose a method for video cataloging by providing metadata 
associated with the digital image, wherein the metadata includes stereotypes of the 
digital image (Figures 16 and 17; Column 13, Lines 52-67, Column 14, Lines 1-31). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al invention according to the teachings of 
Jain et al to provide metadata associated with the digital image, wherein the metadata 
includes stereotypes of the digital image because it will accurately classify and 
intelligently extract information, termed metadata, about the contents of video stream in 
real time (Jain et al, Column 1 , Lines 46-49). 

With regards to Claim 31 , arguments analogous to those presented for Claim 14 
are applicable to Claim 31 . 

With regards to Claim 32, arguments analogous to those presented for Claim 15 
are applicable to Claim 32. 

With regards to Claim 33, arguments analogous to those presented for Claim 16 
are applicable to Claim 33. 

With regards to Claim 47, arguments analogous to those presented for Claim 13 
are applicable to Claim 47. 

With regards to Claim 48, arguments analogous to those presented for Claim 14 
are applicable to Claim 48. 
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With regards to Claim 49, arguments analogous to those presented for Claim 15 
are applicable to Claim 49. 

With regards to Claim 50, arguments analogous to those presented for Claim 16 
are applicable to Claim 50. 

With regards to Claim 94-96, arguments analogous to those presented for Claims 
1 , 8, 9 and 14 are applicable to Claims 94-96. 

6. Claims 1 1 , 12, 28, 29, 45, 46, 62-65, 76-79 and 90-93 are rejected under 35 
U.S.C. 103(a) as being unpatentable over Modestino et al further in view of Jain et al 
(US 6,360,234) and Li et al (U.S. 5,930,783). 

Regarding Claim 1 1 , Modestino et al and Jain et al do not explicitly disclose the 
method according to Claim 10, wherein the contextual data comprises information 
generated by one or more separate sources of the information. 

Li et al disclose a semantic and cognition based image retrieval methodology 
comprising analyzing contextual data generated by one or more separate sources of 
information (Figure 1B, Semantic-based Query, Cognition-based Query; Column 12, 
Lines 30-40). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al and Jain et al combination according to 
the teachings of Li et al to analyze contextual data generated by one or more separate 
sources of information because it will expand versatility of the image segmentation and 
classification. It will utilize the advantages of image retrieval based on both image 
semantics and visual examples of the image (Li et al, Column 3, Lines 27-35). 
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With regards to Claims 28 and 45, arguments analogous to those presented for 
Claim 1 1 are applicable to Claims 28 and 45. 

With regards to Claims 63, 77 and 91 , arguments analogous to those presented 
for the corresponding portion in Claim 1 are applicable to Claims 63, 77 and 91 . Li et al 
retrieve the digital image by using a keyword representing a stereotype (Column 12, 
Lines 30-50). 

With regards to Claims 64, 78 and 92, arguments analogous to those presented 
for the corresponding portion in Claim 1 are applicable to Claims 64, 78 and 92. 
Modestino et al, Jain et al and Li et al do not explicitly retrieve the digital image by using 
an icon to represent a stereotype. Utilizing an icon to represent a keyword is extremely 
well known in the art (Official Notice). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al, Jain et al and Li et al combination to 
Utilize an icon for representing a keyword because icons are a significant factor in the 
user- friendliness of graphical user interface that serve as visual mnemonics to allow the 
user to control certain computer actions without having to remember commands or type 
them at the keyboard. 

With regards to Claims 62, 76 and 90, arguments analogous to those 
represented for Claim 64 concerning utilizing an icon is applicable to claims 62, 76 and 
90. 
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With regards to Claims 65, 79 and 93, arguments analogous to those presented 
for corresponding portion of Claim 1 are applicable to Claims 65, 79 and 93. Li et al 
retrieve the digital image by using a keyword representing a generalization of a 
stereotype (Column 12, Lines 30-50). 

Regarding Claim 12, neither Modestino et al nor Li et al disclose the method 
according to Claim 1 1 , wherein a corresponding portion of said contextual data is 
obtained from a temporal region of interest for each source of said information. 

Jain et al disclose a method for video cataloging by providing metadata 
associated with the image wherein a corresponding portion of contextual data is 
obtained from a temporal region of interest for each source of information (Figures 6-9; 
Column 6, Lines 30-67). 

It would have been obvious to a person of ordinary skill in the art at the time the 
invention was made to modify Modestino et al and Li et al combination according to the 
teachings of Jain et al to obtain a corresponding portion of the contextual data from a 
temporal region of interest for each source of information because it will accurately 
classify and intelligently extract information, termed metadata, about the contents of 
video stream in real time (Jain et al, Column 1, Lines 46-49). 

With regards to Claims 29 and 46, arguments analogous to those presented for 
Claim 12 are applicable to Claims 29 and 46. 

Contact Information 

7. Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Mehrdad Dastouri whose telephone number is (703) 
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305-2438. The examiner can normally be reached on Monday to Friday from 8:00 a.m. 
to 4:30 p.m. If attempts to reach the examiner by telephone are unsuccessful, the 
examiner's supervisor, Amelia Au can be reached on (703) 308-6604. 

The fax phone numbers for the organization where this application or proceeding 
is assigned are (703) 872-9306 for regular and for After Final communications. 

Any inquiry of a general nature or relating to the status of this application or 
proceeding should be directed to the Technology Center Customer Service Office 
whose telephone number is (703) 306-0377. 



Mehrdad Dastouri 
Primary Examiner 
Group Art Unit 2623 
March 6, 2005 
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